User interface device, and projector device

ABSTRACT

A user interface device for detecting an operation, by a finger of a user, on an operation member presented on a projection surface includes a distance detector for detecting a distance to the projection surface, and a distance to the finger, and a controller for detecting the operation based on the distances detected by the distance detector. When presence of the finger between the projection surface and the distance detector is determined, the controller calculates a normal vector of the projection surface based on distances from the distance detector to positions of at least three points on a surface of the projection surface and a distance from the distance detector to the finger, and detects, based on the normal vector, presence or absence of an operation on the operation member.

BACKGROUND

1. Technical Field

The present disclosure relates to a user interface device for detectingan operation on an operation member, and a projector device.

2. Description of the Related Art

Unexamined Japanese Patent Publication No. 2012-104096 (hereinafterreferred to as PTL 1) discloses a projector for projecting an image ofan operation icon on a projection surface, and for allowing a useroperation on the projected operation icon. This projector measuresdistance X in a normal direction from the surface where the projector isplaced to the position of a fingertip of a user based on captured imagedata, and determines whether distance X in the normal direction is equalto or less than a predetermined distance. In the case where distance Xin the normal direction is equal to or less than a predetermineddistance, the projector changes a size of an operation icon immediatelybelow the fingertip to a larger size and projects the icon. Accordingly,even when there is a finger, the influence of the finger can beeliminated, and an operation icon may be made easily visible to the userand a desirable operation is enabled.

PTL 1 discloses a configuration of a projector where it is assumed thata camera which is distance measurement means and a projection surfacesquarely face each other (an optical axis of the camera is orthogonal tothe projection surface). If the camera and the projection surface do notsquarely face each other but obliquely face each other, there is aproblem that the distance is erroneously measured and a user operationis not appropriately detected, and thus, that appropriate behavior as auser interface device may not be realized.

SUMMARY

The present disclosure provides a user interface device capable ofappropriately detecting an operation on a presented operation member,which is a target of a touch operation, even when an object presentingthe operation member and a distance detector do not squarely face eachother, and a projector device.

According to a first mode of the present disclosure, there is provided auser interface device for detecting an operation, by a second object, onan operation member presented on a first object. The user interfacedevice includes a distance detector for detecting a distance to thefirst object, and a distance to the second object, and a controller fordetecting the operation based on the distances detected by the distancedetector. When presence of the second object between the first objectand the distance detector is determined, the controller calculates anormal vector of the first object based on distances from the distancedetector to positions of at least three points on a surface of the firstobject and a distance from the distance detector to the second object,and detects, based on the normal vector, presence or absence of anoperation done by the second object on the operation member.

According to a second mode of the present disclosure, there is provideda projector device including a picture projection unit for projecting apicture of a predetermined operation member on a first object, and auser interface device for detecting an operation on the operation memberby a second object. The user interface device includes a distancedetector for detecting a distance to the first object, and a distance tothe second object, and a controller for detecting the operation based onthe distances detected by the distance detector. When presence of thesecond object between the first object and the distance detector isdetermined, the controller calculates a normal vector of the firstobject based on distances from the distance detector to positions of atleast three points on a surface of the first object and a distance fromthe distance detector to the second object, and detects, based on thenormal vector, presence or absence of an operation done by the secondobject on the operation member.

According to the present disclosure, even in a case where an objectpresenting an operation member as a target of a touch operation and adistance detector do not squarely face each other, the distance betweenthe operation member and an object (for example, a finger of a user)that performs an operation on the operation member may be accuratelydetected. Accordingly, there may be provided a user interface devicecapable of appropriately detecting an operation on an operation memberthat is presented, and a projector device.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram describing a state where a projector device isprojecting a picture on a wall;

FIG. 2 is a diagram describing a state where the projector device isprojecting a picture on a table;

FIG. 3 is a block diagram showing an electrical configuration of theprojector device;

FIG. 4A is a block diagram showing an electrical configuration of adistance detector;

FIG. 4B is a diagram for describing distance information acquired by thedistance detector;

FIG. 5 is a block diagram showing an optical configuration of theprojector device;

FIG. 6A is a diagram for describing a problem that may occur in a casewhere the distance detector and a projection surface, which is a targetof a touch operation, do not squarely face each other;

FIG. 6B is a diagram for describing a problem that may occur in a casewhere the distance detector and the projection surface, which is atarget of a touch operation, do not squarely face each other;

FIG. 7 is a flow chart showing a process for calculating a normal vectorand a distance in a normal direction from a finger to a projectionsurface;

FIG. 8 is a diagram for describing calculation of a distance in thenormal direction between a finger and a projection surface;

FIG. 9 is a diagram for describing three points located around a finger;

FIG. 10 is a diagram for describing determination of a touch operation;

FIG. 11 is a flow chart showing a process for deciding whether anapplication should be executed or not;

FIG. 12 is a diagram for describing example execution of a firstapplication;

FIG. 13A is a diagram for describing example execution of a secondapplication;

FIG. 13B is a diagram for describing example execution of the secondapplication;

FIG. 13C is a diagram for describing example execution of the secondapplication; and

FIG. 14 is a diagram for describing detection of a user operation on aposition, on a projection surface, facing a finger at an angle shiftedfrom a normal direction of the projection surface.

DETAILED DESCRIPTION

Hereinafter, exemplary embodiments will be described in detail withreference to the drawings as appropriate. However, unnecessarilydetailed description may be omitted. For example, detailed descriptionof already well-known matters and repeated description of substantiallythe same structure may be omitted. All of such omissions are intended tofacilitate understanding by those skilled in the art by preventing thefollowing description from becoming unnecessarily redundant.

Moreover, the applicant provides the appended drawings and the followingdescription for those skilled in the art to fully understand the presentdisclosure, and does not intend the subject described in the claims tobe limited by the appended drawings and the following description.

First Exemplary Embodiment

Projector device 100 will be described as a specific exemplaryembodiment of a device on which a user interface device according to thepresent disclosure is mounted.

An outline of a picture projection behavior of projector device 100 willbe described with reference to FIGS. 1 and 2. FIG. 1 is a diagramdescribing a state where projector device 100 is projecting a picture ona wall. FIG. 2 is a diagram describing a state where projector device100 is projecting a picture on a table.

As shown in FIGS. 1 and 2, projector device 100 is fixed to casing 120via drive unit 110. Wires that are electrically connected to respectiveunits of projector device 100 and drive unit 110 are connected with apower source via casing 120 and wire duct 130. Power is thus supplied toprojector device 100 and drive unit 110. Projector device 100 includesan opening 101. Projector device 100 projects a picture on a projectionsurface via opening 101.

Drive unit 110 is capable of driving projector device 100 in such a wayas to change a projection direction of a picture. That is, drive unit110 includes a mechanism for changing the direction of projection device100, and a motor and an actuator for driving the mechanism. Drive unit110 may drive projector device 100 such that the projection direction ofa picture is in the direction of wall 140, as shown in FIG. 1.

Projector device 100 may thereby project picture 141 on wall 140. Driveunit 110 may, in a similar manner, drive projection device 100 such thatthe projection direction of a picture is in the direction of table 150,as shown in FIG. 2. Projector device 100 may thereby also projectpicture 151 on table 150. Drive unit 110 may perform driving based on amanual operation of a user, or may automatically perform drivingaccording to a detection result of a predetermined sensor. Also, thecontents of picture 141 to be projected on wall 140 and picture 151 tobe projected on table 150 may be different or the same.

User interface device 200 is mounted on projector device 100. Userinterface device 200 allows a user to operate the projection surface(wall 140, table 150) itself of a picture (141, 151) as operation meanssuch as a touch panel.

Hereinafter, a configuration and a behavior of projector device 100 willbe described in detail.

1. Configuration of Projector Device

FIG. 3 is a block diagram showing an electrical configuration ofprojector device 100. Projector device 100 includes user interfacedevice 200, light source unit 300, image generator 400, and projectionoptical system 500. In the following, a configuration of each unitconfiguring projection device 100 will be sequentially described.

1.1 User Interface Device

User interface device 200 includes controller 210, memory 220, anddistance detector 230. User interface device 200 is means for detectingan operation when the operation is performed, by a user using his/herfinger 160 as pointing means, on an operation target which is presentedon a predetermined object, and for causing a behavior according to theoperation to be performed.

Controller 210 is a semiconductor device for controlling entireprojector device 100. That is, controller 210 controls the behavior ofeach unit (distance detector 230, memory 220) configuring user interfacedevice 200, and of light source unit 300, image generator 400, andprojection optical system 500. Controller 210 may be configured byhardware only, or may be realized by combining hardware and software.For example, controller 210 may be configured by a CPU, an MPU, an ASIC,an FPGA, or a DSP.

Memory 220 is a storage device storing various pieces of information.Memory 220 is configured by a flash memory, a ferroelectric memory orthe like. Memory 220 stores control programs for controlling projectordevice 100 (including user interface device 200), for example. Also,memory 220 stores various pieces of information supplied by controller210.

Distance detector 230 is configured by a TOF (Time-of-Flight) sensor,for example, and linearly detects the distance to a facing surface. Whendistance detector 230 faces wall 140, the distance from distancedetector 230 to wall 140 is detected. Likewise, when distance detector230 faces table 150, the distance from distance detector 230 to table150 is detected.

FIG. 4A is a block diagram showing an electrical configuration ofdistance detector 230. As shown in FIG. 4A, distance detector 230 isconfigured from infrared light source unit 231 for radiating infrareddetection light, and infrared light receiving unit 232 for receivinginfrared detection light reflected by a facing surface. Infrared lightsource unit 231 radiates infrared detection light through opening 101 insuch a way that the light is diffused to the surrounding. For example,infrared light source unit 231 uses infrared light of a wavelengthbetween 850 nm to 950 nm as the infrared detection light. Infrared lightreceiving unit 232 includes a plurality of pixels that aretwo-dimensionally arranged on an imaging surface.

Controller 210 receives, from infrared light source unit 231,information about a phase of infrared detection light radiated byinfrared light source unit 231, and stores the information in memory220. In the case where a facing surface is inclined or shaped and is notentirely at an equal distance from distance detector 230, the pluralityof pixels that are two-dimensionally arranged on the imaging surface ofinfrared light receiving unit 232 receive reflected light at differenttimings. Since the pixels receive light at different timings, the phaseof the infrared detection light received by infrared light receivingunit 232 is different for each pixel. Controller 210 receives, frominfrared light receiving unit 232, information about the phase of theinfrared detection light received at each pixel of infrared lightreceiving unit 232, and stores the information in memory 220.

Controller 210 reads, from memory 220, the phase of infrared detectionlight radiated by infrared light source unit 231, and the phase ofinfrared detection light received at each pixel of infrared lightreceiving unit 232. Controller 210 measures, based on the phasedifference between the infrared detection light radiated by distancedetector 230 and the infrared detection light received by distancedetector 230, the distance from distance detector 230 to the facingsurface (each pixel), and generates and outputs distance information(distance image) based on the measurement result.

FIG. 4B is a diagram for describing the distance information acquired bydistance detector 230 (infrared light receiving unit 232). Distancedetector 230 detects the distance for each pixel forming an infraredimage that is based on received infrared detection light. Controller 210may thus obtain, on a per pixel basis, a distance detection result forthe entire angle of view of an infrared image received by distancedetector 230. As shown in FIG. 4B, in the following description, thehorizontal direction of an infrared image is taken as the X axis, andthe vertical direction is taken as the Y axis. Moreover, the detecteddistance direction is taken as the Z axis. Controller 210 may acquirethe coordinates (x, y, z) of three axes, X, Y and Z, for each pixelforming the infrared image, based on the detection result of distancedetector 230. That is, controller 210 may acquire the distanceinformation (distance image) based on the detection result of distancedetector 230.

In the present exemplary embodiment, a TOF sensor is cited as an exampleof distance detector 230, but the present disclosure is not limited tosuch an example. That is, distance detector 230 may project a knownpattern such as a random dot pattern and calculate the distance based onthe shift in the pattern, or may use the parallax of a stereo camera.

1.2 Optical Configuration

Next, an optical configuration (picture projection unit) of theprojector device, that is, configurations of light source unit 300,image generator 400 and projection optical system 500, will be describedwith reference to FIG. 5. FIG. 5 is a block diagram showing an opticalconfiguration of projector device 100. As shown in FIG. 5, light sourceunit 300 supplies light that is necessary for generation of a projectionimage to image generator 400. Image generator 400 supplies a generatedpicture to projection optical system 500. Projection optical system 500performs optical transformation such as focusing or zooming on thepicture supplied by image generator 400. Projection optical system 500faces opening 101, and the picture is projected from opening 101.

First, the configuration of light source unit 300 will be described. Asshown in FIG. 5, light source unit 300 includes semiconductor laser 310,dichroic mirror 330, λ/4 plate 340, phosphor wheel 360 and the like.Semiconductor laser 310 is a solid light source that emits S-polarizedblue light having a wavelength of 440 nm to 455 nm, for example.S-polarized blue light that is emitted from semiconductor laser 310enters dichroic mirror 330 via light guiding optical system 320.

For example, dichroic mirror 330 is an optical device having a highreflectance of 98% or more for the S-polarized blue light having awavelength of 440 nm to 455 nm, but having a high transmittance of 95%or more for P-polarized blue light having a wavelength of 440 nm to 455nm and green to red light having a wavelength of 490 nm to 700 nmregardless of the polarization state. Dichroic mirror 330 reflects theS-polarized blue light emitted by semiconductor laser 310 in thedirection of λ/4 plate 340.

λ/4 plate 340 is a polarizer for converting linear polarization intocircular polarization, or for converting circular polarization intolinear polarization. λ/4 plate 340 is disposed between dichroic mirror330 and phosphor wheel 360. S-polarized blue light which has entered λ/4plate 340 is converted into blue light of circular polarization, and isradiated on phosphor wheel 360 through lens 350.

Phosphor wheel 360 is a flat aluminum plate that is capable of rotatingat a high speed. A plurality of B regions which are regions of diffuselyreflecting surfaces, G regions where phosphor that emits green light isapplied, and R regions where phosphor that emits red light is appliedare formed on the surface of phosphor wheel 360. Circular-polarized bluelight radiated on the B region of phosphor wheel 360 is diffuselyreflected, and enters λ/4 plate 340 again as circular-polarized bluelight. Circular-polarized blue light which has entered λ/4 plate 340 isconverted into P-polarized blue light, and enters dichroic mirror 330again. The blue light entering dichroic mirror 330 at this time isP-polarized, and thus the light passes through dichroic mirror 330, andenters image generator 400 via light guiding optical system 370.

Blue light that is radiated on the G region of phosphor wheel 360excites the phosphor applied on the G region, and causes green light tobe emitted. The green light emitted from the G region enters dichroicmirror 330. The green light entering dichroic mirror 330 at this timepasses through dichroic mirror 330, and enters image generator 400 vialight guiding optical system 370. In the same manner, blue light that isradiated on the R region of phosphor wheel 360 excites the phosphorapplied on the R region, and causes red light to be emitted. The redlight emitted from the R region enters dichroic mirror 330. The redlight entering dichroic mirror 330 at this time passes through dichroicmirror 330, and enters image generator 400 via light guiding opticalsystem 370.

Since phosphor wheel 360 is rotating at a high speed, blue light, greenlight, and red light are emitted from light source unit 300 to imagegenerator 400 in a time-division manner.

Image generator 400 generates a projection image according to a picturesignal that is supplied by controller 210. Image generator 400 includesDMD (Digital-Mirror-Device) 420 and the like. DMD 420 is a displaydevice having a large number of micromirrors arranged on the flatsurface. DMD 420 deflects each of the arranged micromirrors according tothe picture signal supplied by controller 210, and spatially modulatesthe entering light. Light source unit 300 emits blue light, green light,and red light in a time-division manner. DMD 420 repeatedly receives,via light guiding optical system 410, blue light, green light, and redlight that are emitted in a time-division manner. DMD 420 deflects eachmicromirror in synchronization with the timing of emission of light ofeach color. Image generator 400 thereby generates a projection imageaccording to the picture signal. DMD 420 deflects the micromirrorsaccording to the picture signal to light that proceeds to the projectionoptical system and light that proceeds to outside the effective coverageof the projection optical system. Image generator 400 may thereby supplythe generated projection image to projection optical system 500.

Projection optical system 500 includes optical members 510 such as zoomlens and focus lens. Projection optical system 500 magnifies the lightentering from image generator 400, and projects the same on a projectionsurface.

A configuration according to a DLP (Digital-Light-Processing) methodusing DMD 420 is described above as an example of projector device 100,but the present disclosure is not limited thereto. That is, projectordevice 100 may alternatively adopt a configuration according to a liquidcrystal method.

Also, a configuration according to a single panel method where a lightsource using phosphor wheel 360 is used in a time-division manner isdescribed above as an example of projector device 100, but the presentdisclosure is not limited thereto. That is, projector device 100 mayadopt a three panel method where various light sources for blue, greenand red colors are provided.

A configuration is described above according to which a light source forblue light for generating a projection picture and a light source forinfrared light for measuring a distance are separate units, but thepresent disclosure is not limited thereto. That is, a light source forblue light for generating a projection picture and a light source forinfrared light for measuring a distance may be an integrated unit. Inthe case of adopting the three panel method, the light sources ofrespective colors and a light source for infrared light may be anintegrated unit.

2. Behavior of User Interface Device

A behavior of user interface device 200 mounted on projector device 100will be described.

Projector device 100 according to the present exemplary embodimentprojects a picture of an operation member, such as an icon, a button ora keyboard, on a projection surface (for example, wall 140 or table 150)where a picture is to be projected. User interface device 200 detects anoperation, on an operation member projected on projection surface 180,performed by a user using his/her finger 160 as pointing means, forexample. Then, user interface device 200 causes a behavior according tothe detected operation to be performed.

First, a problem that is assumed with respect to user interface device200 that performs above-described behavior will be described. FIGS. 6Aand 6B are diagrams for describing a problem that may occur in a casewhere distance detector 230 and projection surface 180, which is atarget of a touch operation, do not squarely face each other. FIGS. 6Aand 6B show a state where projector device 100 is inclined and does notsquarely face facing projection surface 180 (for example, wall 140 ortable 150). In such a case, there is a problem that the distance betweenthe projection surface and an object (for example, a finger of a user)performing an operation on the projection surface is erroneouslymeasured, and that a user operation is not appropriately detected.

FIGS. 6A and 6B each show a case where finger 160 of a user ispositioned at position PA1 and a case where finger 160 is positioned atposition PA2. Here, normal distance D′1 from projection surface 180 toposition PA1 and normal distance D′2 from projection surface 180 toposition PA2 are the same. Also, it is assumed that a user who isplacing finger 160 at position PA1 intends a touch operation on positionPB1 which is immediately below and which is on projection surface 180.In the same manner, it is assumed that a user who is placing finger 160at position PA2 intends a touch operation on position PB2 which isimmediately below and which is on projection surface 180.

As described above, distance detector 230 linearly detects the distanceto facing projection surface 180. Accordingly, distance detector 230detects, as the distance to finger 160 positioned at position PA1,distance D1 from the distance detector to position PAL In the samemanner, distance detector 230 detects distance D4 as the distance tofinger 160 positioned at position PA2.

A method is conceivable according to which determination of a touchoperation on projection surface 180 by finger 160 of a user is realizedby using a detection result of distance detector 230 (distance fromdistance detector 230 to projection surface 180) as it is, and bycomparing the distance with a predetermined threshold. A touch operationmay be appropriately determined by this method if distance detector 230and projection surface 180 squarely face each other (if a lightreceiving surface of infrared light receiving unit 232 of distancedetector 230 and projection surface 180 are parallel with each other).

However, as shown in FIGS. 6A and B, in the case where distance detector230 of projector device 100 and projection surface 180 do not squarelyface each other but obliquely face each other, even if the normaldistance from finger 160 to projection surface 180 is the same, thedetection result of distance detector 230 (distance from distancedetector 230 to projection surface 180) greatly varies depending on theposition of finger 160 performing a touch operation. For example, asshown in FIG. 6A, the normal distances (D′1, D′2) to projection surface180 are actually the same for a case where finger 160 is positioned atposition PA1 and a case where finger 160 is positioned at position PA2,but determined distances D3 and D6 to be used for touch operations aregreatly different.

Regarding specific determined distances, in the case where finger 160 isat position PA1, the distance from distance detector 230 to finger 160is D1, and the distance from distance detector 230 to the projectionsurface is D2 (distance to the projection surface is acquired in advanceby distance detector 230). Accordingly, the distance from finger 160 toprojection surface 180 is D3=(D2−D1). On the other hand, in the casewhere finger 160 is at position PA2, the distance from distance detector230 to finger 160 is D4, and the distance from distance detector 230 toprojection surface 180 is D5. Accordingly, the distance from finger 160to projection surface 180 is D6=(D5−D4), and the determined distance isgreatly different for a case where finger 160 is positioned at positionPA1 and where finger 160 is positioned at position PA2.

The inventor(s) of the present invention has/have test-manufactured andstudied user interface device 200 which causes, in a case where distancedetector 230 and projection surface 180 do not squarely face each otherbut obliquely face each other, projection surface 180 to function as asurface which is to be taken as a target of a touch operation. As aresult, the inventor(s) of the present invention has/have come torecognize the present problem in the course of test-manufacturing andstudying user interface device 200 and to devise means for solutiondescribed below.

Specifically, the inventor(s) of the present invention has/have deviseda method, as shown in FIG. 6B, for detecting a normal vector (normaldistance) from finger 160 to projection surface 180 regardless of theposition of finger 160, and for determining a touch operation based onthe detected normal vector. User interface device 200 which mayappropriately determine a touch operation even in a case where distancedetector 230 and projection surface 180 do not squarely face each otherbut obliquely face each other is thus realized.

In the following, a configuration for detecting a normal vector (normaldistance) from finger 160 to projection surface 180 regardless of theposition of finger 160, and for determining a touch operation based onthe detected normal vector will be described in detail.

FIG. 7 is a flow chart showing a process for calculating a normal vectorfrom finger 160 to projection surface 180. FIG. 8 is a diagram fordescribing calculation of the distance in the normal direction betweenfinger 160 and projection surface 180. FIG. 9 is a diagram fordescribing three points located around finger 160. Additionally, in thepresent exemplary embodiment, a picture showing a user interface (icon,button, keyboard, etc.) to be used by a user for an operation isprojected by user interface device 200 on projection surface 180.

First, controller 210 acquires distance information from distancedetector 230 (S700). Next, controller 210 detects, based on the acquireddistance information, whether an object (a finger of a user, forexample) has entered between distance detector 230 and projectionsurface 180 where a picture as a target of a touch operation isprojected (S710). Any known technology may be used for decisionregarding entering of an object. For example, whether an object hasentered or not may be decided by using temperature detection by infraredradiation, such as by a presence sensor. Alternatively, whether anobject has entered or not may be decided by a motion detection processof calculating a time difference (inter-frame difference) of imagesobtained by capturing the projection surface as a target of a touchoperation, and of determining whether the difference has exceeded aspecific threshold or not.

Controller 210 repeats step S700 of acquiring distance information fromdistance detector 230 (S700, NO in S710) until entering of an objectbetween projection surface 180 and distance detector 230 is detected. Onthe other hand, in the case where entering of an object betweenprojection surface 180 and distance detector 230 is detected (YES inS710), controller 210 acquires, from memory 220, distance informationfor projection surface 180 before current detection of entering of anobject (S720). Additionally, pieces of distance information for severalpast frames including an immediately preceding frame are stored inmemory 220 as information about the distance to projection surface 180.For example, the distance to projection surface 180 is measured bydistance detector 230 in an initialization process at the time ofturning on of projector device 100, in a state where there is no objectbetween projection surface 180 and distance detector 230, and distanceinformation is stored in memory 220 based on the measurement result.

Next, controller 210 detects whether the object detected in step S710 isfinger 160 or not (S730). Whether it is a finger or not may be detectedby performing a matching process based on information indicating thefeatures of the shape of a finger stored in advance, for example.Alternatively, the matching process may be performed based on distanceinformation for a finger stored in advance. Moreover, any knowntechnology may be used in the method for detecting whether an object isa finger or not.

In the case where the object which has entered is detected to be otherthan a finger (NO in S730), controller 210 returns to the process instep S700, and acquires distance information.

In the case where the object which has entered is detected to be afinger (YES in S730), controller 210 acquires distance information (XYZcoordinates) of at least three points, on projection surface 180,present around the position (XY-coordinate position) of the detectedfinger (S740). Specifically, as shown in FIG. 8, controller 210 obtainsposition (XYZ coordinates) PA of the finger from the distanceinformation for projection surface 180 which has been stored in memory220 in step S720. Next, as shown in FIG. 9, controller 210 acquiresdistance information for between distance detector 230 and projectionsurface 180 for positions PD1 to PD3 of the at least three points nearposition PA obtained.

Subsequently, controller 210 calculates normal vector 190 of projectionsurface 180 from the pieces of distance information (XYZ coordinates) ofthe three points acquired in step S740 (S750). Specifically, controller210 calculates orthogonal vectors of two vectors represented by thethree points, by calculating the cross product of the two vectorsrepresented by three points based on the pieces of distance information(XYZ coordinates) of the three points. A unit vector of the calculatedorthogonal vectors is normal vector 190 of projection surface 180.

Then, controller 210 calculates, from normal vector 190 calculated instep S750 and the position of the finger calculated in step S740, thedistance (normal distance) from the coordinates of the finger to a pointof intersection of a line segment, extending in the normal direction ofthe projection surface, and the projection surface (S760). Specifically,the normal distance is obtained by calculating an absolute value of aninner product of a vector connecting the position (XYZ coordinates) ofthe finger calculated in step S740 and the position (XYZ coordinates),on the projection surface, corresponding to the XY-coordinate positionof the finger, and normal vector 190 of the projection surfacecalculated in step S750. For example, in FIG. 8, normal distance D′ isobtained by calculating an absolute value of an inner product of avector connecting position PA of the finger and position PB, onprojection surface 180, corresponding to position PA of the finger, andnormal vector 190 of projection surface 180.

Then, controller 210 stores the normal distance calculated in step S760in memory 220 (S770). In the case where a normal distance is alreadystored in memory 220, controller 210 overwrites (updates) the normaldistance by the newly calculated normal distance.

Additionally, when entering of an object is no longer detected in stepS710, or when an entering object is detected in step S730 to be otherthan a finger, controller 210 erases the normal distance that is storedin memory 220.

Then, controller 210 returns to step S700, and repeats the processdescribed above.

The normal distance from finger 160 of the user to projection surface180 is obtained in the above manner, and is stored in memory 220. Thenormal distance obtained in the above manner is used for determinationof a finger of a user coming close to a picture (icon, button, etc.) foroperation that is projected on the projection surface.

3. Execution of Application Based on Operation

Hereinafter, a process for deciding whether an application based on anoperation on an interface image that is projected on projection surface180 should be executed or not will be described. FIG. 10 is a diagramfor describing determination of a touch operation. Here, a case isconsidered, as shown in FIG. 10, where interface image 185 (icon,button, keyboard, etc.) as an operation target is displayed onprojection surface 180. In the case of performing a touch operation oninterface image 185, a user brings his/her finger 160 close to interfaceimage 185.

FIG. 11 is a flow chart showing a behavior for deciding whether anapplication based on a user operation on the interface image should beexecuted or not. Controller 210 decides whether execution of anapplication should be carried out or not, by performing touchdetermination based on normal distance D′ between finger 160 andprojection surface 180.

Specifically, controller 210 compares normal distance D′ stored inmemory 220 with a predetermined threshold (S800). In the case wherenormal distance D′ is at or above the predetermined threshold (NO inS800), controller 210 determines that a touch operation is notperformed, and does not execute an application. On the other hand, inthe case where normal distance D′ is below the predetermined threshold(YES in S800), controller 210 determines that a touch operation has beenperformed, and executes a predetermined application corresponding tointerface image 185 on which the touch operation has been performed(S810). Controller 210 repeats the process described above at apredetermined cycle.

3.1 Example Execution of Application

Next, example execution of an application based on determination of atouch operation by projector device 100 will be described. FIG. 12 is adiagram for describing example execution of a first application(application that is started by a touch operation). FIGS. 13A, 13B and13C are diagrams for describing example execution of a secondapplication (application that is started by a gesture operation).

Control regarding determination of a touch operation according to theexemplary embodiment described above may be used with respect to anapplication for interacting with a picture that is projected onprojection surface 180 (for example, wall 140 or table 150) by projectordevice 100. That is, in the case where it is determined that a touchoperation is performed, controller 210 may switch a picture to beprojected according to the coordinate position where the touch operationis performed. For example, as shown in FIG. 12, in the case wherepictures 51, 52 and 53 are projected on projection surface 180, if atouch operation is determined to have been performed on one of thepictures, controller 210 switches the picture to be projected accordingto picture 51, 52 or 53 which has been touched. In the example shown inFIG. 12, switching is performed between pictures 61, 62 and 63. In otherwords, control regarding determination of a touch operation according tothe present exemplary embodiment may be applied to an application forshifting to a picture that is set in advance, according to coordinateswhere a touch operation is determined to have been performed. As anexample of such an application, an application is conceivable whichprojects, on projection surface 180, a picture of a menu of a restaurantshowing a plurality of names of dishes, and which displays, when a nameof a dish is selected, an image or detailed information about the dishcorresponding to the name of the dish.

Furthermore, a second application has a function of changing a projectedpicture according to a so-called gesture operation. FIGS. 13A and 13Bare diagrams describing a function of changing a projected imageaccording to a pinch-in or pinch-out operation. That is, controller 210enlarges or reduces a picture projected by projector device 100 onprojection surface 180 based on a change in the distance betweencoordinates of two points where touch operations are performed. In thiscase, touch determination has to be performed for two points, and theabove-described determination control may be used in the determinationof presence or absence of a touch operation on a plurality of points.

FIG. 13C is a diagram describing a function of changing a projectedimage according to a drag-and-drop operation. As shown in FIG. 13C, inthe case where a user performs a touch operation on a projected pictureand moves the position of the touch operation while maintaining a statewhere the touch operation is performed (while maintaining the height ofthe finger above projection surface 180), a picture which is the targetof the touch operation may be moved according to the movement of theposition of the touch operation. In the case of such an application, itis desirably determined that a touch operation is performed when thedistance in the normal direction between finger 160 of a user andprojection surface 180 becomes as small as possible. For example,considering the average thickness of a person's finger and a variance inthe accuracy of a sensor of distance detector 230, a predeterminedthreshold to be compared against the normal distance may be made about30 mm. Additionally, depending on the accuracy of the sensor of distancedetector 230 to be used, the threshold may be further reduced.

4. Effects, Etc.

As described above, projector device 100 according to the presentexemplary embodiment includes a picture projection unit for projectinginterface image 185, which is a picture of a predetermined operationmember, on projection surface 180 (example of a first object), and userinterface device 200 for detecting an operation on the operation memberby finger 160 (example of a second object) of a user. User interfacedevice 200 includes distance detector 230 for detecting the distance toprojection surface 180 and the distance to finger 160 of the user, andcontroller 210 for detecting an operation based on the distancesdetected by distance detector 230. When determining that finger 160 ispresent between projection surface 180 and distance detector 230,controller 210 calculates normal vector 190 of projection surface 180based on distances from distance detector 230 to positions of at leastthree points on a surface of projection surface 180 and the distancefrom distance detector 230 to finger 160, and detects an operation oninterface image 185 by finger 160 based on normal vector 190.

For example, controller 210 may calculate distance D′, in the normaldirection of projection surface 180, which is the distance betweenfinger 160 and projection surface 180, based on calculated normal vector190, and may detect an operation on the operation member when calculateddistance D′ is within a predetermined range.

According to the configuration described above, even in a case whereprojection surface 180 that presents an operation member, which is atarget of a touch operation, and distance detector 230 do not squarelyface each other, the distance between the operation member and an object(such as a finger of a user) for performing an operation on theoperation member may be accurately detected, and thus an operation onthe presented operation member may be appropriately detected.Accordingly, an application may be appropriately executed according to auser operation.

OTHER EXEMPLARY EMBODIMENTS

The first exemplary embodiment has been described above as an example ofthe technology disclosed in the present application.

However, the technology in the present disclosure is not limited to theabove exemplary embodiment, and may also be applied to exemplaryembodiments which have been subjected to modifications, substitutions,additions, or omissions as required. Moreover, it is also possible tocombine the structural elements described in the first exemplaryembodiment to realize a new exemplary embodiment. In the following,other exemplary embodiments will be described as examples.

The above-described exemplary embodiment describes an example where userinterface device 200 is applied to projector device 100, but the presentdisclosure is not limited to such a case. That is, operation means whichare to be targets of operation by a user are represented by projectedpictures, but these operation means may also be provided by othermethods. For example, the operation means may be presented by beingdisplayed on a sheet of paper where menus or the like are drawn, or onanother display device such as a liquid crystal display. Theconfiguration of user interface device 200 described above may beadopted as a user interface for such operation means presented in theabove manner. Moreover, the configuration of user interface device 200described above may be adopted as an interface of a control device thatperforms a process when a touch operation is performed not on aprojected picture but an object in the real world itself.

An example is described above where the distance (D′) in the normaldirection is used in the determination of a touch operation, but thepresent disclosure is not limited to such an example. FIG. 14 is adiagram for describing detection of a user operation at a position, onthe projection surface, facing the finger at an angle shifted from thenormal direction of the projection surface. As shown in FIG. 14,distance (D1′, D2′) between detected position PA of a finger andposition (PB1′, PB2′) having offset angle (θ) from the normal directionmay be used in the determination of presence or absence of a touchoperation instead of using distance D′ itself in the normal direction inthe determination of a touch operation. Also in this case, normal vector190 has to be calculated to calculate distance (D1′, D2′). Specifically,normal vector 190 from projection surface 180 to finger 160 iscalculated, and normal distance D′ is obtained based on calculatednormal vector 190, and distance (D1′, D2′) to position (PB1′, PB2′)having offset angle (θ) may be calculated based on normal distance D′and the offset angle. A predetermined behavior may then be performedbased on distance (D1′, D2′).

Moreover, the distance to a position of one point, on projection surface180, having the same XY coordinates as the detected position of finger160 may be used in the determination of a touch operation instead ofusing the distance itself in the normal direction in the determinationof a touch operation. In this case, memory 220 of user interface device200 stores a distance correction table according to inclination (α) ofprojection surface 180 with respect to distance detector 230. Thedistance correction table stores, in association with each other, aninclination (α) of projection surface 180 with respect to distancedetector 230 and a correction value for a threshold. Then, a correctionvalue for a threshold may be read from the distance correction tablebased on the position of detected finger 160 and the inclination ofprojection surface 180, and a predetermined threshold used in thedetermination of a touch operation may be corrected based on thecorrection value for the threshold.

Specifically, controller 210 calculates the distance between finger 160and projection surface 180 based on the calculated position (XYZcoordinates) of finger 160 and the position (XYZ coordinates), onprojection surface 180, corresponding to the XY-coordinate position ofthe finger. Also, controller 210 acquires pieces of distance information(XYZ coordinates) for at least three points, on projection surface 180,around the XY-coordinate position of the finger. That is, controller 210acquires pieces of distance information for at least three pointsbetween distance detector 230 and projection surface 180. Then,controller 210 calculates normal vector 190 of projection surface 180based on the XYZ coordinates indicated by the acquired pieces ofdistance information for the three points. Controller 210 calculates theinclination between distance detector 230 and projection surface 180from calculated normal vector 190. Controller 210 refers to anappropriate correction value in the distance correction table stored inmemory 220, based on the calculated inclination and the XYZ coordinatesof finger 160 detected by distance detector 230. Then, controller 210corrects the predetermined threshold based on the correction value whichhas been referred to, and uses the threshold in the determination of atouch operation. Appropriate determination of a touch operation may thusbe performed by referring to the distance correction table.Additionally, also in the exemplary embodiment where the distancecorrection table is referred to, user interface device 200 calculatesnormal vector 190 of the projection surface as in the exemplaryembodiment described above.

Moreover, the exemplary embodiment described above cites wall 140 andtable 150 as examples of a facing surface (projection surface 180), butthese are not restrictive. Projector device 100 or user interface device200 according to the present exemplary embodiment may adopt an object ofany shape as the facing surface (projection surface). User interfacedevice 200 may appropriately determine that a touch operation isperformed even with an object of an arbitrary shape.

The exemplary embodiments have been described above as examples of thetechnology in the present disclosure. The appended drawings and thedetailed description have been provided for this purpose.

Therefore, the structural elements shown in the appended drawings anddescribed in the detailed description include not only structuralelements that are essential for solving the problem but also otherstructural elements in order to illustrate the technology. Hence, thatthese non-essential structural elements are shown in the appendeddrawings and described in the detailed description does not cause thesestructural elements to be immediately recognized as being essential.

Furthermore, since the exemplary embodiments described above are forillustrating the technology in the present disclosure, variousmodifications, substitutions, additions, and omissions may be performedwithin a range of claims and equivalents to the claims.

The present disclosure may provide a user interface device that iscapable of appropriately detecting a user operation. The user interfacedevice of the present disclosure may be applied not only to theprojector device as described above, but also to various devices fordetecting an operation on an operation member according to a detectionresult of a distance detector.

What is claimed is:
 1. A user interface device for detecting anoperation, by a second object, on an operation member presented on afirst object, the user interface device comprising: a distance detectorfor detecting a distance to the first object, and a distance to thesecond object; and a controller for detecting the operation based on thedistances detected by the distance detector, wherein, when presence ofthe second object between the first object and the distance detector isdetermined, the controller calculates a normal vector of the firstobject based on distances from the distance detector to positions of atleast three points on a surface of the first object and a distance fromthe distance detector to the second object, and detects, based on thenormal vector, presence or absence of an operation done by the secondobject on the operation member.
 2. The user interface device accordingto claim 1, wherein the controller calculates, based on the calculatednormal vector, a distance, in a normal direction of the first object,between the second object and the first object, and when the calculateddistance is within a predetermined range, the controller detects that anoperation is done by the second object on the first object.
 3. The userinterface device according to claim 1, wherein, when an operation doneby the second object on the operation member is detected, the controllercauses a predetermined behavior according to the operation to beperformed.
 4. The user interface device according to claim 1, whereinthe operation member is provided as a picture that is projected on thefirst object.
 5. The user interface device according to claim 1, whereinthe operation member is provided by being directly drawn on the firstobject, or by placing a member, on which the operation member is drawn,on the first object.
 6. A projector device comprising; a pictureprojection unit for projecting a picture of a predetermined operationmember on a first object; and a user interface device for detecting anoperation done by a second object on the operation member, wherein theuser interface device includes; a distance detector for detecting adistance to the first object, and a distance to the second object, and acontroller for detecting the operation based on the distances detectedby the distance detector, wherein, when presence of the second objectbetween the first object and the distance detector is determined, thecontroller calculates a normal vector of the first object based ondistances from the distance detector to positions of at least threepoints on a surface of the first object and a distance from the distancedetector to the second object, and detects, based on the normal vector,presence or absence of an operation done by the second object on theoperation member.